1. Technical Field of the Invention
This invention pertains to managing denial of service (DOS) attacks. More particularly, it relates to denying service to clients issuing more than a configurable number of outstanding (incompleted) negotiation requests.
2. Background Art
For Secure Sockets Layer (SSL) client/server applications, it is often the case that an SSL client will attempt to connect to an SSL server application and request a secure SSL connection. For the SSL application server, it is necessary to authenticate such clients using a common certificate store, which must be accessed any time a new SSL session is established. This certificate store contains certificates for those clients that are authorized to the server system and server applications.
As is typical in SSL application server design, application programming interfaces (APIs) are used to access SSL functions. The following SSL functions define a standard sockets API used in TCP/IP networks.                SSL_CREATE( ) enables SSL support for a specified socket descriptor.        SSL_DESTROY( ) ends SSL support for the specified SSL session.        SSL_HANDSHAKE( ) initaties the SSL handshake protocol.        SSL_INIT( ) initializes the current job for SSL.        SSL_READ( ) receives data from an SSL-enabled socket descriptor.        SSL_WRITE( ) writes data to an SSL-enabled socket descriptor.        
Referring to FIG. 1, a server or client application that uses the sockets and SSL APIs contains the following elements:                1. A call to SOCKET( ) 60 or 80 to obtain a socket descriptor.        2. A call to SSL_INIT( ) 62 or 82 to initialize the job environment for SSL processing. An SSL_INIT( ) call must succeed at least once in a job.        3. A call to SSL_CREATE( ) 64 OR 84 to enable SSL support for the connected socket.        4. Socket calls to activate a connection. The application calls CONNECT( ) 86 to activate a connection for a client program, or it calls BIND( ) 68, LISTEN( ) 70 and ACCEPT( ) 72 to activate a connection for a server program.        5. A call to SSL_HANDSHAKE( ) 96 or 98 to initiate the two way SSL handshake negotiation 130, 132 of the cryptographic parameters. Both a server program and the client programs with which it communicates must provide a certificate for an SSL handshake 130, 132 to succeed. A server must also provide the private key that is associated with its certificate or its key ring file. The SSL_INIT( ) CALL 62 identifies the key ring file from which the certificate and private key are obtained for all SSL sessions established for a job.        6. Calls to SSL_READ( ) 110 and 118 and SSL_WRITE( ) 116 and 112 to receive and send data 134, 136.        7. A call to SSL_DESTROY( ) 120 or 122 to disable SSL support for the socket.        8. A call to CLOSE( ) 124 or 126 to destroy the connected sockets.        
A problem can exist in the sequence of calls illustrated in FIG. 1 at two way handshake 130, 132. SSL_HANDSHAKE( ) 96, 98 requires two-way communication between the client and server programs to complete successfully. An SSL client program written by a hacker can be written to not complete the client-side handshake 132, and effectively block the SSL server application hostage—preventing further use of that server. The server is blocked because it is obligated to wait for the client-side handshake 132 before it can continue. This is the well-known Internet style attack called ‘denial of service’ or DOS. Its effect is that any new SSL clients that attempt to connect to the application server cannot get a connection, because the blocked server never sees their connect request 86. Also poorly written clients can inadvertently cause DOS attacks to occur. Whatever the case, a particular SSL application server is blocked while waiting for a client to properly complete the SSL handshake 132. If the client never responds, the server never continues to process the request 114.
SSL API developers anticipated a server might block on the call to SSL_HANDSHAKE( ) 96, and therefore supply a timeout setting 88. This means an SSL application server can be written to break out of a blocked state with a client in order to continue. This solution is difficult to implement effectively, since if the timeout value 88 is set too large, a DOS attack is effective for the duration of the timeout period 88. On the other hand, if the timeout period 88 is set too short, network delays can make it hard for a valid SSL client to successfully connect.
Single threaded servers are particularly vulnerable to denial of service attacks because total blocking of all processes or functions of the server may be the result.
It is an object of the invention to provide an improved system and method for managing denial of service attacks.
It is a further object of the invention to provide a system and method for reducing the impact of a denial of service attack on server resources.
It is a further object of the invention to provide a system and method for managing denial of service attacks which does not unduly restrict service to legitimate clients.
It is a further object of the invention to deny service to DOS hacker clients and maximize service to legitimate clients.
It is a further object of the invention to provide a system and method for minimizing or eliminating server resources (CPU, storage, etc.) used by malicious clients.